When testing computer programs, it is important to measure the thoroughness of the testing. Testing involves executing compiled computer code to ensure that no errors are generated and to ensure that the program produces the expected result or functionality. If an error is discovered, it may be corrected. However, fixing one error can create a new errors or uncover previously hidden errors. Therefore, each time the code is modified, it needs to be retested.
Code is commonly tested by executing the program and operating the computer program to exercise as much of the code as possible. Tools are available to assist in code testing by identifying what percentage of the code has been executed during the course of a test. Code can be instrumented by existing pre-compilers, or tested using interpreters or just-in-time compilers for monitoring the execution of the program in terms of the source code. For example, the compiled code may be modified by a pre-compiler or compiled with a compile option provided with the compiler that integrates source code coverage reporting functionality in the resulting compiled code. This information can then be used to identify the percentage of statements executed during the test, and perhaps target specific routines or procedures that require additional testing. The percentage of source code that is executed during a test is referred in the art as “code coverage.”
Thus, code coverage analysis is a way to measure the thoroughness of a test by measuring the amount of code executed, either by statement, path, or condition, during a test. Code coverage by statement provides the percentage of statements that were executed. Code coverage by path provides a percentage of possible execution paths that were followed in the course of the test. Code coverage by condition provides a percentage of evaluation points, such as a true/false decision, that were executed. Other types of code coverage are known, such as basic block coverage and function coverage. A basic block is a code segment that is always executed together, i.e., it has no internal jumps or jump targets. A function is a subdivision of code that can be called from other locations in the program. Achieving 100% code coverage in any category can be very time consuming and inefficient.
Unfortunately, testing anything below 100% code coverage is very subjective. For instance, testing a certain percentage of code does not necessarily mean the right code is tested. For example, a fairly exhaustive test scenario may only test 20 percent of the code, yet an incomplete test scenario may actually cover 35% of the code. Code coverage analyses, especially among black box testers, is often looked down upon because of this limitation. The phrase “black box testing” refers to a testing methodology wherein varying inputs are presented to a computer program, and the resulting outputs are checked to ensure that they conform to the expected outputs, without regard to the internal workings of the program.
Because of the inadequacies of existing code coverage measurement methodologies as described above, a new mechanism is needed for estimating a quality of the code coverage in addition to, or instead of, a simple percentage value of code coverage.